Method and Devices for Generating, Transferring and Processing Three-Dimensional Image Data

ABSTRACT

Three-dimensional digital image data comes from a stereographic imaging arrangement ( 501, 502, 1201 ) that takes a first raw image ( 601 ) along a first optical axis and a second raw image ( 602 ) along a second optical axis. The imaging arrangement ( 501, 502, 1201 ) has a maximum imaging depth ( 506 ) and a minimum imaging depth ( 505 ). It transmits the first raw image ( 601 ) and the second raw image ( 602 ) to a receiving device ( 1102 ) along with an indication of a disparity range between a maximum disparity value and a minimum disparity value. The maximum disparity value is a measure of a difference between locations in the first ( 601 ) and second ( 602 ) raw images that represent the minimum imaging depth ( 505 ). The minimum disparity value is a measure of a difference between locations in the first ( 601 ) and second ( 602 ) raw images that represent the maximum imaging depth ( 506 ).

CROSS REFERENCE TO RELATED APPLICATIONS

This application is the U.S. National Stage of International ApplicationNumber PCT/FI2005/000491 filed on Nov. 17, 2005 which was published inEnglish on May 24, 2007 under International Publication Number WO2007/057497.

TECHNICAL FIELD

The invention concerns generally the technology of obtaining,transferring and outputting three-dimensional image data. Especially theinvention concerns the problem of transferring three-dimensional imagedata in a form that allows it to be displayed with any display device.

BACKGROUND OF THE INVENTION

The visual system of the brain produces a perception ofthree-dimensionality by combining the two slightly different imagescoming from the eyes. An image displayed on a two-dimensional displayscreen can give rise to the same perception without the need of specialviewing glasses or the like, if the display screen is autostereoscopic,i.e. in itself capable of emitting slightly different information to theright and left eye of the viewer. The two autostereoscopic displaytechnologies that are most widely used for this purpose at the time ofwriting this specification are known as the parallax barrier principleand the lenticular principle, although also other approaches are knownas well.

FIG. 1 is a simple schematic example of a known parallax barrier typeliquid crystal display. A liquid crystal layer 101 comprises right-eyesubpixels and left-eye subpixels marked with R and L respectively. Abacklighting layer 102 emits light from behind the liquid crystaldisplay. A parallax barrier layer 103 contains slits that only allowlight to propagate through the right-eye subpixels to the right eye ofthe viewer and through the left-eye subpixels to the left eye of theviewer. It is also possible to have the parallax barrier layer 103 infront of the liquid crystal layer 101 instead of between it and thebacklighting layer 102.

FIG. 2 is a simple schematic example of a known lenticular type liquidcrystal display. Also here the liquid crystal layer 201 comprisesright-eye subpixels and left-eye subpixels. The backlighting layer 202emits light through the liquid crystal layer 201. A layer 203 oflenticulars, i.e. cylindrical lenses, collimates the light so that lightrays coming through a right-eye subpixel continue parallelly towards theright eye of the viewer and light rays coming through a left-eyesubpixel continue parallelly towards the left eye of the viewer.

FIG. 3 illustrates schematically a known principle for generatingthree-dimensional image information of a group of imaged objects. Twohorizontally separated cameras 301 and 302 take pictures at the sametime but otherwise independently of each other, resulting in twoso-called raw images 303 and 304 respectively. Images of the objectsappear at different locations in the raw images, because the cameras 301and 302 see the imaged objects from different directions. It should benoted, though, that the differences in the raw images appear in highlyexaggerated proportion in FIG. 3 compared to most practical solutions,because for reasons of making FIG. 3 graphically clear the imagedobjects are drawn very close to the camera arrangement. Together the rawimages 303 and 304 constitute a stereograph that could be displayedusing any suitable display technology, including but not being limitedto those illustrated in FIGS. 1 and 2.

There are no widespread standards that would define the parameters thataffect the generation of stereographs or their presentation on displayscreens. Numerous parameters have a significant effect, such as theseparation between cameras; focal length; size, resolution and angularpixel pitch of the CCD (Charge-Coupled Device) arrays in the cameras;size, resolution and pixel structure of the display; default viewingdistance; and the amount of scaling, cropping and other processing thatis required to map the raw images to the subpixel arrays,time-interlaced fields or other display elements that eventually presentthe fused image to the viewer. The lack of standards means that astereographic image taken with a certain imaging arrangement andprepared for presentation on a particular display type is not likely towork well on any other display type.

The incompatibility problem will become more and more important whenthree-dimensional imaging and autostereoscopic displays find their wayto simple and inexpensive consumer appliances, such as portablecommunication devices, where conventional cameras and high-qualitytwo-dimensional displays are already in widespread use. A user that hastaken a three-dimensional image with a portable communication device ofone brand wants to be sure that he can transmit the image to anotheruser, who can view it correctly irrespective of which brand of a devicethe recipient has.

A US patent publication number US 2004/0218269 A1 discloses a 3D DataFormatter, which acts as a format converter between various knowninterlacing techniques and is also capable of certain basic pictureprocessing operations, such as zooming, cropping and keystonecorrecting. Simply changing between presentation formats does not solvethe problem of inherent incompatibility between displays that may be ofdifferent size and may have a different default viewing distance. Aweakness of the reference publication is also that the solutionconsidered therein can only work between formats and interlacingtechniques that the formatter device knows in advance. The referencepublication does not consider any way of generating good fusible 3Dimage content for any receiving device, the features of which are notyet known.

SUMMARY OF THE INVENTION

An objective of the present invention is to present a method and devicesfor generating, transferring and processing three-dimensional image datain a form that does not require ensuring compatibility between theimaging arrangement and the displaying arrangement in advance. Anotherobjective of the present invention is to enable efficient transfer ofgeneric three-dimensional image content.

The objectives of the invention are achieved by recording a disparityrange that describes limits of how much the raw images differ from eachother, and resealing the disparities related to different viewing depthsto map the three-dimensional image into a comfortable viewing spacebetween the maximum virtual distances in front of and behind the displayat which objects in the image should appear.

A transmitting device according to the invention is characterized by thefeatures recited in the characterizing part of the independent claimdirected to a transmitting device.

An imaging module according to the invention is characterized by thefeatures recited in the characterizing part of the independent claimdirected to an imaging module.

A receiving device according to the invention is characterized by thefeatures recited in the characterizing part of the independent claimdirected to a receiving device.

A transmission system according to the invention is characterized by thefeatures recited in the characterizing part of the independent claimdirected to a transmission system.

A transmitting method according to the invention is characterized by thefeatures recited in the characterizing part of the independent claimdirected to a transmitting method.

A receiving method according to the invention is characterized by thefeatures recited in the characterizing part of the independent claimdirected to a receiving method.

Software program products for a transmission operation and a receptionoperation according to the invention are characterized by the featuresrecited in the characterizing part of the independent claims directedsuch software program products.

Embodiments of the invention are described in the depending claims.

Objects that appear in a three-dimensional image are located at variousdistances from the imaging arrangement. The distance between the imagingarrangement and an imaged object is commonly referred to as the imagingdepth. We may reasonably assume that there are minimum and maximumlimits to imaging depth: for example, all imaged objects must appearbetween half a meter and infinity. The structural parameters of thecamera arrangement determines, what is the disparity between raw imagesrelated to each imaging depth value. The disparities related to theminimum imaging depth and the maximum imaging depth define a disparityrange that is characteristic to each particular imaging arrangement.

The disparity range can be recorded, stored and transmitted togetherwith a pair of raw images. An autostereographic display that is to beused for fusing the raw images into a three-dimensional image has acharacteristic comfortable viewing space, which extends from a virtualfront edge in front of the display screen to a virtual back edge behindthe display screen. By suitably scaling and shifting the disparitiesbetween the raw images it is possible to find new disparities that makethe fused image appear within the comfortable viewing space: objectsthat were located at the maximum depth from the imaging arrangement aremapped to the back edge of the comfortable viewing space, and objectsthat were located at the minimum depth are mapped to the front edge.

Since the limits of the comfortable viewing space depend partly on thepersonal preferences of each user, there should be a possibility ofdynamically modifying them. The invention makes this particularly easy,because how much in front of or behind the plane of the display screenobjects seem to appear depends directly on the corresponding disparitybetween the component images. If the user wants to e.g. shift the frontedge of the comfortable viewing space towards the plane of the displayscreen, he simply tells the displaying device to decrease thecorresponding maximum disparity value.

An important advantage of the invention is its adaptability toautomatically and instantly display images on autostereographic displaysof various sizes. An image may be viewed on a small-sizeautostereographic display of a portable communications device or adisplay screen of a personal computer, or even on a giant screen of a 3Dcinema system. According to the invention, the same transmission format(raw images+disparity range) can be used to transmit a stereographicimage to all these purposes, so that only some disparity mapping isneeded to adapt the image for displaying in each case. The inventionallows flexible content sharing and seamless interaction between allkinds of devices, portable and non-portable, that are capable ofdisplaying stereographic images.

The exemplary embodiments of the invention presented in this patentapplication are not to be interpreted to pose limitations to theapplicability of the appended claims. The verb “to comprise” is used inthis patent application as an open limitation that does not exclude theexistence of also unrecited features. The features recited in dependingclaims are mutually freely combinable unless otherwise explicitlystated.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features which are considered as characteristic of theinvention are set forth in particular in the appended claims. Theinvention itself, however, both as to its construction and its method ofoperation, together with additional objects and advantages thereof, willbe best understood from the following description of specificembodiments when read in connection with the accompanying drawings.

FIG. 1 illustrates a known parallax barrier display principle,

FIG. 2 illustrates a known lenticular display principle,

FIG. 3 illustrates the known principle of producing a stereographicimage

FIG. 4 illustrates the concept of comfortable viewing space,

FIG. 5 illustrates an imaging arrangement imaging a close object and adistant object,

FIG. 6 illustrates a pair of raw images,

FIG. 7 illustrates certain angular relations,

FIG. 8 illustrates mapping the virtual appearance of the close anddistant objects to the comfortable viewing space,

FIG. 9 illustrates subpixel separation associated with the image of adistant object,

FIG. 10 illustrates subpixel separation associated with the image of aclose object,

FIG. 11 illustrates the transmission between a transmitting device, areceiving device and a display,

FIG. 12 illustrates an exemplary composition of functional blocks in thetransmitting device, the receiving device and the display,

FIG. 13 illustrates a method and a software program product executed bya transmitting device, and

FIG. 14 illustrates a method and a software program product executed bya receiving device and a display.

DETAILED DESCRIPTION

FIG. 4 illustrates schematically how the three-dimensional image takenin FIG. 3 could appear to a viewer on an autostereographic displayscreen 401. We assume for simplicity that the imaged objects aretransparent bubbles. In order to correctly display point A in the image,the corresponding right-eye subpixel should appear at AR and thecorresponding left-eye subpixel should appear at AL. Point A is thepoint of the imaged objects that was closest to the camera arrangement,so in the displayed image it appears closest to the viewer. In order tocorrectly display the most distant point B in the image, thecorresponding right-eye and left-eye subpixels should appear at BR andBL respectively.

Point A appears to be virtually located at a distance 402 in front ofthe display screen 401, and point B appears to be virtually located at adistance 403 behind the display screen 401. How large are the distances402 and 403 depends on the disparity between AR and AL as well as BR andBL respectively as well as on the viewing distance (the distance betweenthe eyes of the viewer and the display screen). For reasons of humanvisual ergonomy, it is not possible to stretch the distances 402 and 403more than certain limits that depend on the size of the display screenas well as on the default viewing distance. For example, when looking atpoint A, the viewer's eyes should focus at the distance of the displayscreen 401 but converge on a point that is closer by the amount ofdistance 402, which contradicts the normal rules of operation of thehuman visual system. There are no globally valid maximum values for thedistances 402 and 403 in FIG. 4: what a viewer considers comfortable isafter all a matter of personal taste.

Generally we may define the concept of a comfortable viewing space sothat it extends from the maximum distance in front of the display screenwhere objects of the three-dimensional image may be made to virtuallyappear to the maximum distance behind the display screen where objectsof the three-dimensional image may be made to virtually appear, so thatsaid features of human visual ergonomy still allow said objects to beviewed comfortably. Assuming that the three-dimensional image has beenmade to utilize the whole depth of the comfortable viewing space in FIG.4, we may denote the depth of the comfortable viewing space as distance404. Experiments have shown that for a display screen of a portablecommunications device, the default viewing distance of which is in theorder of 40-60 cm, an upper limit of the disparity between AR and AL—orbetween BR and BL—is in the order of a few millimeters, but dependsremarkably on the exact type and size of the display. At the time ofwriting this description there are portable-device-sized displays inwhich the disparity should not be more than about ±2 mm, but also somein which it can be conveniently about ±10 mm. The plus or minus signcomes from the fact that for close objects the right-eye subpixel shouldbe more to the left than the left-eye subpixel (see AR vs. AL) while fordistant objects the left-eye subpixel should be more to the left thanthe right-eye subpixel (see BL vs. BR).

Features of the display screen have a major effect on how far in frontof the display screen and how far behind the display screen thecomfortable viewing space will reach. Said features include factors likesize and shape of the display; the default viewing distance; the size,shape and distribution of pixels; the sharpness and resolution of thedisplay; reverse half occlusions (resulting from foreground objectsbeing cut by a window that is perceived behind the object) and thestructure and operation of the autostereography mechanism. Thus it ispossible to determine for each displaying device certain default valuesfor distances 402 and 403 that set the comfortable viewing space at adefault location in relation to the display screen. Since the questionof comfortable viewing is ultimately subjective and depends on personaltaste, it is advantageous to allow the default values to be changedaccording to user preferences.

Having defined the comfortable viewing space we consider in more detailthe concept of disparity between left-eye and right-eye subimages. FIG.5 illustrates an imaging arrangement in which two horizontally separatedcameras 501 and 502 each take a picture of a scenery that comprises avery close object 503 and a very distant object (or background) 504. Weassume that minimum and maximum imaging depths have been defined for theimaging arrangement, and that the close object 503 happens to be exactlyat the minimum imaging depth 505 while the distant object 504 happens tobe at the maximum imaging depth 506. For excluding possible ambiguities,it is useful to define the minimum imaging depth 505 and the maximumimaging depth 506 along a central axis of the imaging arrangement. Itshould also be noted that the minimum imaging depth 505 and the maximumimaging depth 506 are features of the imaging arrangement and notfeatures of any particular image, even if in the exemplary case of FIG.5 the objects 503 and 504 happen to be located at exactly the minimumand maximum imaging depths respectively.

The optical axes of the cameras 501 and 502 are parallel to each otherin FIG. 5, which means that the central axis mentioned above is animaginary line that is parallel to the optical axes and located in themiddle between them. It has been found that using parallel camerasrather than converged ones, the optical axes of which would intersect atsome default imaging depth, produces superior image quality. Since themaximum imaging depth has been set at infinity in FIG. 5, mathematicallyspeaking this is the same as using converged cameras with just theoptical axis intersection point taken to infinity. Thus, nocontradiction is caused by saying that the optical axes of the cameras501 and 502 intersect at the maximum imaging depth. An importantconsequence thereof is that an object 504 located at the maximum imagingdepth 506 has zero disparity in the raw images 601 and 602 illustratedin FIGS. 6 a and 6 b. In other words, the distant object appears at thesame horizontal location in each raw image. To be very exact, there is asmall but finite horizontal difference because even a distant object isnever truly at infinite distance, but for practical considerations wemay neglect the small difference for the time being.

Other values than zero for minimum disparity are possible either so thatthe cameras are converged, meaning that objects that are more distantthan the intersecting point of the optical axes will have a negativedisparity, or so that for some reason related to e.g. lighting orfocusing possibilities it is practical to define maximum imaging depthto be considerably less than infinity. In the last-mentioned case theminimum disparity will have a small positive value.

The close object 503 does not appear at the same horizontal location inthe raw images. In the left-hand raw image 601 it appears X/2 units tothe right from the center of the raw image. Since the close object 503was centered on the imaginary central axis between the optical axes ofthe cameras, in the right-hand raw image it appears correspondingly X/2units to the left from the center. The close object 503 was located atthe predefined minimum imaging depth 505, so we may say that thedisparity X between its appearances in the raw images 601 and 602 is themaximum disparity. All disparity values associated with intermediateobjects would fall between the maximum disparity X and the minimumdisparity 0, which latter value could be denoted with Y. Speaking inangular terms, X/2 which is one half of the maximum disparitycorresponds to an angular separation between the optical axis of acamera and a line drawn from the camera to a centrally located object atthe minimum imaging depth.

The purpose is to map the eventually resulting three-dimensional imageto the comfortable viewing space of a display so that the closestobjects in the image will virtually appear in front of the displayscreen and the most distant objects will virtually appear behind thedisplay screen. How should one decide, what object (if any) shouldvirtually appear exactly in the plane of the display screen? Generallyspeaking that could be decided quite freely, but FIGS. 7 and 8illustrate at least one alternative that holds for displays for whichthe maximum absolute value of disparity is the same for objectsappearing virtually in front of the screen and objects appearingvirtually behind the screen.

The basic principle is that objects that were exactly at the minimumimaging depth should virtually appear exactly at the front edge of thecomfortable viewing space, and objects that were exactly at the maximumimaging depth should virtually appear exactly at the back edge of thecomfortable viewing space. We may draw an imaginary line at half waybetween the optical axis (i.e. the direction to the centrally locatedobject at the maximum imaging depth) and the direction to the centrallylocated object at the minimum imaging depth for both cameras. Theselines intersect at imaging depth 701 in FIG. 7. Similarly in thedisplayed three-dimensional image of FIG. 8, lines drawn exactly in themiddle of the angle between the directions to the center of the frontedge and the center of the back edge of the comfortable viewing spaceintersect exactly at the plane of the display screen 401. We may deducethat if the imaged scenery contained an object at imaging depth 701,that object would virtually appear in the plane of the display screen.Basic trigonometry could be used to derive an exact formula for theimaging depth 701, defined in terms of camera separation and the minimumand maximum imaging depths.

Concerning displays for which the maximum absolute value of disparity isnot the same for objects appearing virtually in front of the screen andobjects appearing virtually behind the screen, similar geometricconsiderations can be made. For example, if the absolute value of themaximum disparity for objects that virtually appear behind the screen is3 mm and the absolute value of the maximum disparity for objects thatvirtually appear in front of the screen is 2 mm, instead of the simplehalf-way angle lines of FIGS. 7 and 8 we should draw lines in the middlethat in each case divide the angle between the limiting directions intocomponent angles that have the relative magnitudes of 3:2 instead of 1:1as in FIGS. 7 and 8. The zero disparity plane would then be at thedistance where these drawn lines intersect.

FIGS. 9 and 10 illustrate defining the disparities for the most distantobject and closest object in the displayed image respectively. In FIG.9, to make an object virtually appear at the back edge of thecomfortable viewing space, there should be a disparity the absolutevalue of which is Y′ units between the left-eye and right-eye subpixelsassociated with said object. In FIG. 10, to make an object virtuallyappear at the front edge of the comfortable viewing space, there shouldbe a disparity the absolute value of which is X′ units. We must notethat the sign of this disparity is opposite to the sign of the disparityassociated with the most distant object. Selecting the signs of thedisparities is just a matter of convention. Here we define thatdisparities where the left-eye subpixel is to the left of the right-eyesubpixel are negative, and disparities where the left-eye subpixel is tothe right of the right-eye subpixel are positive. If we make thisselection, we must note that in the imaging arrangement of FIGS. 5 and 6all disparities will have positive values (i.e. all objects closer thaninfinity will appear in the right-eye raw image more to the left than inthe left-eye raw image).

In order to correctly map the image taken with the imaging arrangementof FIGS. 5 and 6 to the displaying arrangement of FIG. 8, we should thusconstruct a disparity mapping function that

-   -   maps a disparity X between the raw images into a disparity X′        between subpixels in the displayed image    -   maps a disparity 0 (or more generally: a disparity associated        with objects at the maximum imaging depth) between the raw        images into a disparity −Y′ between subpixels in the displayed        image and    -   linearly maps all disparities between the limiting values X and        0 between the raw images into corresponding disparities between        the limiting values X′ and −Y′ between subpixels in the        displayed image.

Linearly mapping a range of values into another range of values is asimple mathematical operation that only requires a scaling factor and adisplacement value.

When handling digital images, the most natural unit of disparity ispixels. However, we must note that in general, the imaging arrangementhas a different resolution and thus a different number of pixels inhorizontal direction across the image than the displaying arrangement.This has to be taken into account in determining the scaling factor.Assuming that the maximum disparity (i.e. the disparity associated withthe front edge of the comfortable viewing space) and the minimumdisparity (i.e. the disparity associated with the back edge of thecomfortable viewing space) of the displaying arrangement are D_(out,max)and D_(out,min) respectively, and that the maximum disparity (i.e. thedisparity associated with the minimum imaging depth) and the minimumdisparity (i.e. the disparity associated with the maximum imaging depth)of the imaging arrangement are D_(in,max) and D_(in,min) respectively,the most natural selection for the scaling factor SF is

SF=(D _(out,max) −D _(out,min))/(D _(in,max) −D _(in,min))  (1)

and the most natural selection for the displacement value DV is

DV=(D _(out,min) D _(in,max) −D _(out,max) D _(in,min))/(D _(in,max) −D_(in,min)).  (2)

Alternatively we may express the amount ZD of how much the zerodisparity plane should be displaced as

ZD=(D _(in,max) +D _(in,min) −D _(out,max) −D _(out,min))/2.  (3)

As an example, an imaging arrangement might have D_(in,max)=+60 pixelsand D_(in,min)=0 pixels, and a displaying arrangement might haveD_(out,max)=+10 pixels and D_(out,min)=−10 pixels. Applying formulas (1)and (2) above, we get values SF=⅓ and DV=−10, so for any arbitrarydisparity D_(in) in the raw images we get the corresponding disparityD_(out) in the displayed image as

D_(out)=⅓*D _(in)−10.  (4)

A concept of displacing the zero disparity plane without scaling is thesame as performing a mapping from input disparities to intermediatedisparities, assuming that the number of pixels in horizontal directionacross the image remains the same. Concerning processing order, it ispossible to first displace the zero disparity plane without scaling andto thereafter scale the intermediate disparities into output disparitiesto take into account the different number of pixels in horizontaldirection across the image. The other possibility is to scale first anddisplace the zero disparity plane thereafter. Of these twopossibilities, the first-mentioned tends to produce more accurateresults.

To be mathematically exact, we must note that the simple linear modelabove is an approximation, because exactly speaking the half-way anglebetween the direction to the closest possible object and the directionto the most distant possible object does not divide the correspondingwidth across the display into half. However, the difference between thelinear approximation and the exact, sinusoidal relationship is so smalltaken the small angles that are involved in practice that it can beneglected.

For the purposes of the present invention it is important to note thatthe maximum and minimum disparities D_(in,max) and D_(in,min) that mayoccur in a pair of raw images does not depend on image content but onlyon the properties of the imaging arrangement. Similarly the maximum andminimum disparities D_(out,max) and D_(out,min) that correspond toobjects virtually appearing at the front and back edge of thecomfortable viewing space do not depend on image content but only on theproperties of the displaying arrangement. Naturally relatively fewimages will actually include objects at the very minimum or the verymaximum imaging depth but something in between, but it will then be onthe responsibility of the displaying arrangement to find thecorresponding disparity values that will fall between the extremelimits.

For the purposes of the present invention it should also be noted thatthe actual process of interlacing, in which the displaying arrangementdetects the pixels that represent close or distant objects andconsequently associates each pixel pair in the raw images with theappropriate disparity value, is not important to the invention. Severalknown algorithms exist for comparing the raw images and finding thepixels that represent the same point in the imaged scenery although theyappear horizontally displaced in the raw images. The present inventionconcerns the question of how does the displaying arrangement define themapping function that maps an input disparity, once found, to an outputdisparity that will determine the horizontal distance between theleft-eye and right-eye subpixels.

FIG. 11 illustrates a data flow process according to an embodiment ofthe invention when three-dimensional digital image data is transferredfrom a transmitting device 1101 to a receiving device 1102 andsubsequently displayed on a display 1103 coupled to the receiving device1102. FIG. 12 illustrates an example of certain functional blocks ofsaid devices that may take part in preparing, transferring, processingand outputting the image data.

The transmitting device 1101 transmits the raw images and an indicationof an associated disparity range to the receiving device. In the exampleof FIG. 12 we assume that the transmitting device is also the originatorof the three-dimensional image data, for which purpose it comprises atleast two parallel cameras 1201. The raw images taken by the cameras arestored in an image memory 1202. Characteristics of the imagingarrangement, such as camera properties, camera separation, general imageproperties, possible standardized minimum and maximum imaging depth arestored in a characteristics memory 1203. A control means (controller)1204 is provided in the transmitting device for enabling a user tocontrol the operations of taking, handling and transmittingthree-dimensional images. For transmitting raw images and the associateddisparity ranges to outside of the transmitting device there is providedoutput means (output module) 1205, which in the case of a portablecommunications device typically include a wireless transceiver. Thecameras 1201, the image memory 1202 and the characteristics memory 1203or a part of these functional blocks could be implemented as an imagingmodule that can be manufactured and sold separately for installing tovarious kinds of electronic devices.

In the simplest possible case the imaging characteristics of thetransmitting device are constant: the cameras are fixedly located in thetransmitting device, they have a fixed focal length, the minimum andmaximum imaging depths are constant and so on. In that case it isparticularly simple to store indications of the maximum and minimumdisparity values associated with a pair of raw images, because also saidmaximum and minimum disparity values will be constant. However, it ispossible that the cameras are equipped with zoom objectives orexchangeable lenses that change the focal length, or the aperture orseparation between cameras can be changed, or there may be more than twocameras so that the user may select which of them to use, or some otherimaging characteristic is not constant. For such cases it is useful tohave a coupling from the control means 1204 to the characteristicsmemory 1203 so that whatever changes the user makes to the imagingarrangement, always the most appropriate corresponding indications ofthe maximum and minimum disparity values may be read from thecharacteristics memory 1203 and transmitted along with the raw images.The coupling from the control means 1204 to the characteristics memory1203 may be also indirect, so that when the user changes focal length orother imaging characteristic, the imaging arrangement produces anindication of the change that causes a subsequent read operation in thecharacteristics memory to find the most appropriate information.

The nature and content of the indication of the maximum and minimumdisparity values may also vary. The most straightforward alternative isto express the maximum and minimum disparity values in the units ofpixels that have one-to-one correspondence with the horizontal pixelcount in the raw images, and transmit the explicit values along with theraw images. It is also possible to use some other units than pixels.Another alternative is that the transmitting device already compares theraw images enough to find the pixel pairs that correspond to the closestimage point and the most image point; then the transmitting device couldspecifically identify these pixels in the raw images rather thanannounce any measure of their separation. Yet another alternative may beused if the imaging characteristics of the transmitting device areconstant. The maximum and minimum disparity values typical to eachcommonly known type of transmitting device could be standardized, sothat the transmitting device only needed to transmit an indication ofits type. A receiving device could then consult some previously storedlook-up table that associates imaging device types with theircharacteristic maximum and minimum disparity values. Yet anotheralternative is that the transmitting device transmits geometric detailsabout the imaging arrangement along with the raw images, such as themaximum and minimum imaging depths, focal length, CCD size and/orothers, from which a receiving device in turn can derive the appropriatemaximum and minimum disparity values of the original imagingarrangement. Yet another alternative is to define that the minimumdisparity between the raw image is always zero or some other constantvalue, so that no indication of minimum disparity needs to betransmitted. Constantly assuming a zero minimum disparity is synonymousto assuming that the two optical axes always intersect at the maximumimaging depth, which may but does not have to be infinity.

Whatever is the nature and content of the indication of the maximum andminimum disparity values, the receiving device 1102 receives it alongwith the raw images through a receiver 1211. The raw image data goes toan image processor 1212, where it waits to be processed while adisparity mapper 1213 prepares the mapping between disparity values inthe raw images and disparity values for the interlaced images to bedisplayed. In order to perform the mapping, the disparity mapper 1213must know the disparity range associated with the raw images, as well asthe allowable disparity range that may appear eventually in thedisplayed image. If the latter is constant, the disparity mapper 1213may simply read it from a characteristics memory 1214. Otherwise thedisparity mapper 1213 may calculate the disparity range allowable in thedisplayed image from stored information such as display size, displaypixel pitch, and human ergonomic factors of the display (includingdefault viewing distance). If the receiving device can be coupled to avariety of displays, it is advisable to arrange storing the appropriate,display-dependent values to the characteristics memory 1214 at the timewhen a new display is coupled. Most advantageously the receiving device1102 comprises also control means 1215 through which a user may inputhis preferences about increasing or decreasing the disparity rangeallowable in the displayed image.

Once the disparity range allowable in the displayed image is known, thedisparity mapper 1213 may use it and its knowledge about the originaldisparity range associated with the raw images to produce the mappingfunction (see e.g. formulas (1)-(4) above), which it delivers to theimage processor 1212. The task of the image processor 1212 is ultimatelyto convert the raw images into the new image pair that will be conveyedto the display 1103. This converting may include scaling and cropping ofthe images as well as making the changes to disparity pixel-wise.Cropping is needed because displacing the zero disparity planeeffectively moves the background of each raw image sideways so thatinformation is lost from a vertical bar at each side edge of the image.This vertical bars must be cut out. Also in many cases the proportionsof the raw images are not the same as the proportions of the display,which means that either the image must be cropped to fit to the displayor empty fields must be added to some sides of the image.

As we have pointed out earlier, for the present invention it is notimportant, what algorithm the image processor 1212 uses to identifymutually corresponding pixels in the raw images. The invention affectsthe changes to be made in disparity: once the image processor 1212 hasfound a pixel pair with some initial disparity D_(in) in the raw images,it relocates the pixels of that pixel pair in the new images so thattheir new disparity is calculated according to the mapping formula thatis based on knowing the initial disparity range as well as the disparityrange allowable in the displayed image.

The display 1103 is shown to comprise the interlacing means (interlacingmodule) 1221 that directs the new images prepared in the image processor1212 to the arrays of subpixels that constitute the display screen 1222.

FIGS. 13 and 14 illustrate the methods performed at the transmitting andreceiving ends respectively. The drawings may also be considered asillustrations of the software program products employed at thetransmitting and receiving ends. At step 1301 the transmitting devicerecords the raw images, and at step 1302 it obtains an indication of thedisparity range, i.e. the maximum and minimum disparity valuesassociated with the raw images. At step 1303 the transmitting devicecombines the raw images and the indication of the disparity range fortransmission, and at step 1304 it transmits them to a receiving device.

The receiving device receives the raw images and the indication of thedisparity range associated with the raw images at step 1401. It obtainsinformation about the display characteristics at step 1402 and uses thatinformation to determine the disparity range allowable in the displayedimage at step 1403. At step 1404 the receiving device uses theinformation it has about the raw image disparity range and the outputimage disparity range to determine the appropriate disparity mapping. Atstep 1405 the images are processed, which includes relocating the pixelsin the horizontal direction according to the disparity mapping functiondetermined in the previous step. Depending on what display technologyand image file standard is to be used, the processing step 1405 mayinclude format conversion operations such as those explained for examplein the prior art publication US 2004/0218269 A1. For example, if theimage is to be displayed on a parallax barrier display of SharpElectronics Corporation, the left-eye and right-eye subimages arecompressed in horizontal direction by a factor 2 and written side byside into a single image file that follows otherwise the JPEG (JointPhotographic Experts Group) standard but has a certain additionalcharacter string in its header and an extension .stj in its file name.At 1406 the completed new images are output on the display.

The exemplary embodiments described above should not be construed aslimiting. For example, even if we have consistently considered usingonly two cameras, the invention does not exclude using three or morecameras. In a multi-camera arrangement the concept of disparity must bereplaced with a more generally defined horizontal displacement of pixelsdepending on imaging depth, but otherwise the principle of the inventioncan be applied in a straightforward manner. For each camera there can bedefined the characteristic horizontal displacements associated withobjects at the minimum imaging depth and the maximum imaging depth onthe central line of sight of the imaging arrangement. Thesecharacteristic horizontal displacements take the position of maximum andminimum disparities associated with a raw image pair in the descriptionabove. Also even if we have considered solely still images so far, itshould be noted that the principle of the invention is also applicableto the obtaining, transmitting, processing and displaying of series ofimages, which as a sequence constitute a video signal. Since the initialdisparity range is a property of the imaging arrangement and does notdepend on image content, applying the invention to the processing of avideo signal is particularly simple: the indication of the initialdisparity range only needs to be transmitted once unless thecharacteristics of the imaging arrangement are dynamically changedduring shooting, in which case the indication of the disparity rangeneeds to be transmitted regularly so that it covers each change.

Yet another exemplary modification concerns the implementation of thecameras: instead of the (at least) two parallel cameras it is possibleto equip a single camera (i.e. a single CCD array) with a branching lenssystem that sets up two parallel optical paths and includes a shuttersystem that allows taking a picture through each optical path in turn.Additionally, even if we have used the term “raw image” to generallydescribe an image taken along one optical path and transmitted to areceiving device, this does not mean that the transmitted image shouldbe “raw” in the sense that it would not have undergone any changes orprocessing after having read from the CCD array. Normal image processingcan be applied, like color and brightness correction, computedcorrections to remove undesired optical effects, and the like. However,care must be taken not to delete information that is important to thereconstruction of a stereographic image. For example, image compressionaccording to the JPEG format may average out adjacent pixels, whichmeans that if such image compression were to be applied at aninappropriate phase of handling stereographic images, it might destroythe stereographic properties altogether.

The numeric values used in the description are exemplary. For example,even if at the time of writing the description the default viewingdistance of the autostereographic displays of portable devices is in theorder of 40-60 cm, other displays may involve shorter or longer defaultviewing distances. Displays for use in personal computers have usuallylonger default viewing distances, up to 1-2 meters. Much longer defaultviewing distances occur in audiovisual presentation systems like 3Dcinemas.

1. A device comprising a stereographic imaging arrangement configured totake a first raw image along a first optical axis and a second raw imagealong a second optical axis, wherein: the first and second optical axeshave a separation at the imaging arrangement, and the imagingarrangement has a maximum imaging depth and a minimum imaging depth;wherein: the device is configured to transmit the first raw image andthe second raw image to a receiving device along with an indication of adisparity range between a maximum disparity value and a minimumdisparity value, the maximum disparity value is a measure of adifference between a location in the first raw image that represents theminimum imaging depth and a location in the second raw image thatrepresents the minimum imaging depth, and the minimum disparity value isa measure of a difference between a location in the first raw image thatrepresents the maximum imaging depth and a location in the second rawimage that represents the maximum imaging depth.
 2. The device accordingto claim 1, wherein in order to transmit said indication of a disparityrange, the device is configured to transmit a maximum disparity inpixels between a location in the first raw image that represents theminimum imaging depth and a location in the second raw image thatrepresents the minimum imaging depth.
 3. The device according to claim2, wherein the device is further configured to transmit a minimumdisparity in pixels between a location in the first raw image thatrepresents the maximum imaging depth and a location in the second rawimage that represents the maximum imaging depth.
 4. The device accordingto claim 1, wherein in order to transmit said indication of a disparityrange, the device is configured to transmit an identifier of at leastone of a type of the device or a type of the imaging arrangement.
 5. Thedevice according to claim 1, wherein the device comprises: a controllerfor changing optical characteristics of the imaging arrangement and acharacteristics memory for outputting said indication of a disparityrange for transmission; wherein said characteristics memory isconfigured to respond to changes made to the optical characteristics ofthe imaging arrangement by outputting an indication of a disparity rangethat corresponds to the changes made.
 6. The device according to claim1, wherein it is a portable communications device and comprises anoutput module configured to transmit the first raw image, the second rawimage and the indication of the disparity range to the receiving devicethrough a wireless communications network.
 7. An imaging modulecomprising: a stereographic imaging arrangement configured to take afirst raw image along a first optical axis and a second raw image alonga second optical axis; wherein: the imaging module is configured tostore the first raw image and the second raw image and an indication ofa disparity range between a maximum disparity value and a minimumdisparity value, the maximum disparity value is a measure of adifference between a location in the first raw image that represents aminimum imaging depth and a location in the second raw image thatrepresents the minimum imaging depth, wherein said minimum imaging depthis a feature of said imaging arrangement and the minimum disparity valueis a measure of a difference between a location in the first raw imagethat represents a maximum imaging depth and a location in the second rawimage that represents the maximum imaging depth, wherein said maximumimaging depth is a feature of said imaging arrangement.
 8. The imagingmodule according to claim 7, wherein in order to store said indicationof a disparity range, the imaging module is configured to store amaximum disparity in pixels between a location in the first raw imagethat represents the minimum imaging depth and a location in the secondraw image that represents the minimum imaging depth.
 9. The imagingmodule according to claim 8, wherein the imaging module is furtherconfigured to store a minimum disparity in pixels between a location inthe first raw image that represents the maximum imaging depth and alocation in the second raw image that represents the maximum imagingdepth.
 10. The imaging module according to claim 7, wherein in order tostore said indication of a disparity range, the imaging module isconfigured to store an identifier of at least one of a type of theimaging module or a type of the imaging arrangement.
 11. An imagingmodule comprising: a stereographic imaging arrangement configured totake a first raw image along a first optical axis and a second raw imagealong a second optical axis; wherein: the imaging module is configuredfor storing the first raw image and the second raw image and configuredfor obtaining and storing an indication of a disparity range between amaximum disparity value and a minimum disparity value, the maximumdisparity value is a measure of a difference between a location in thefirst raw image that represents a minimum imaging depth and a locationin the second raw image that represents the minimum imaging depth,wherein said minimum imaging depth is a feature of said imagingarrangement and the minimum disparity value is a measure of a differencebetween a location in the first raw image that represents a maximumimaging depth and a location in the second raw image that represents themaximum imaging depth, wherein said maximum imaging depth is a featureof said imaging arrangement.
 12. The imaging module according to claim11, wherein in order to store said indication of a disparity range, theimaging module is configured for storing a maximum disparity in pixelsbetween a location in the first raw image that represents the minimumimaging depth and a location in the second raw image that represents theminimum imaging depth.
 13. The imaging module according to claim 12,wherein the imaging module further is configured for storing a minimumdisparity in pixels between a location in the first raw image thatrepresents the maximum imaging depth and a location in the second rawimage that represents the maximum imaging depth.
 14. The imaging moduleaccording to claim 11, wherein in order to store said indication of adisparity range, the imaging module is configured for storing anidentifier of at least one of a type of the imaging module or a type ofthe imaging arrangement.
 15. A device comprising a receiver; wherein:the device is configured to receive from a transmitting device a firstraw image and a second raw image along with an indication of a disparityrange between a maximum input disparity value and a minimum inputdisparity value, said maximum input disparity value is a measure of adifference between a location in the first raw image that represents aminimum imaging depth of an imaging arrangement and a location in thesecond raw image that represents the minimum imaging depth, and theminimum disparity value is a measure of a difference between a locationin the first raw image that represents a maximum imaging depth of theimaging arrangement and a location in the second raw image thatrepresents the maximum imaging depth.
 16. The device according to claim15, wherein as said indication of a disparity range between a maximuminput disparity value and a minimum input disparity value, the device isconfigured to receive a maximum disparity in pixels between a locationin the first raw image that represents the minimum imaging depth and alocation in the second raw image that represents the minimum imagingdepth.
 17. The device according to claim 16, wherein the device isfurther configured to receive a minimum disparity in pixels between alocation in the first raw image that represents the maximum imagingdepth and a location in the second raw image that represents the maximumimaging depth.
 18. The device according to claim 15, wherein: as saidindication of a disparity range between a maximum input disparity valueand a minimum input disparity value, the device is configured to receivean identifier of at least one of a type of the transmitting device or atype of an imaging arrangement used to produce the first and second rawimages, and the device is configured to derive, from the receivedindication, a maximum disparity in pixels between a location in thefirst raw image that represents the minimum imaging depth and a locationin the second raw image that represents the minimum imaging depth and aminimum disparity in pixels between a location in the first raw imagethat represents the maximum imaging depth and a location in the secondraw image that represents the maximum imaging depth.
 19. The deviceaccording to claim 15, wherein: the device is configured to store amaximum output disparity value and a minimum output disparity value, ofwhich the maximum output disparity value is a measure of a difference inlocation between two output subpixels that represent a virtual distancein front of a display screen, and the minimum output disparity value isa measure of a difference in location between two output subpixels thatrepresent a virtual distance behind the display screen and the device isconfigured to perform a linear mapping from input disparities in thefirst and second raw images that are between the maximum input disparityvalue and the minimum input disparity value to output disparities thatare between the maximum output disparity value and the minimum outputdisparity value, in order to find locations for output subpixels to bedisplayed on an autostereographic display.
 20. The device according toclaim 19, wherein it comprises a controller configured to convert inputinformation given by a user into changes of at least one of the maximumoutput disparity value and the minimum output disparity value.
 21. Thedevice according to claim 15, wherein the device comprises anautostereographic display for displaying a three-dimensional imagederived from the first and second raw images and the indication of thedisparity range.
 22. An image transmission system comprising atransmitting device and a receiving device, of which the transmittingdevice comprises a stereographic imaging arrangement configured to takea first raw image along a first optical axis and a second raw imagealong a second optical axis, and the receiving device comprises areceiver, wherein: the transmitting device and the receiving device areconfigured to exchange the first raw image and the second raw image andan indication of a disparity range between a maximum disparity value anda minimum disparity value, the maximum disparity value is a measure of adifference between a location in the first raw image that representsminimum imaging depth of the imaging arrangement and a location in thesecond raw image that represents the minimum imaging depth, and theminimum disparity value is a measure of a difference between a locationin the first raw image that represents a maximum imaging depth of theimaging arrangement and a location in the second raw image thatrepresents the maximum imaging depth.
 23. The image transmission systemaccording to claim 22, wherein in order to exchange said indication of adisparity range, the transmitting device and the receiving device areconfigured to exchange a maximum disparity in pixels between a locationin the first raw image that represents the minimum imaging depth and alocation in the second raw image that represents the minimum imagingdepth.
 24. The image transmission system according to claim 23, whereinthe transmitting device and the receiving device are further configuredto exchange a minimum disparity in pixels between a location in thefirst raw image that represents the maximum imaging depth and a locationin the second raw image that represents the maximum imaging depth. 25.The imaging module according to claim 22, wherein in order to exchangesaid indication of a disparity range, the transmitting device and thereceiving device are configured to exchange an identifier of at leastone of a type of the imaging module or a type of the imagingarrangement.
 26. A method for transmitting three-dimensional digitalimage data, comprising: transmitting a first raw image taken along afirst optical axis, transmitting a second raw image taken along a secondoptical axis; and transmitting an indication of a disparity rangebetween a maximum disparity value and a minimum disparity value, whereinthe maximum disparity value is a measure of a difference between alocation in the first raw image that represents a minimum imaging depthof an imaging arrangement used to produce the first and second rawimages and a location in the second raw image that represents theminimum imaging depth, and the minimum disparity value is a measure of adifference between a location in the first raw image that represents amaximum imaging depth of the imaging arrangement and a location in thesecond raw image that represents the maximum imaging depth.
 27. Themethod according to claim 26, comprising transmitting a maximumdisparity in pixels between a location in the first raw image thatrepresents the minimum imaging depth and a location in the second rawimage that represents the minimum imaging depth, in order to transmitsaid indication of a disparity range.
 28. The method according to claim27, further comprising transmitting a minimum disparity in pixelsbetween a location in the first raw image that represents the maximumimaging depth and a location in the second raw image that represents themaximum imaging depth.
 29. The method according to claim 26, comprisingtransmitting an identifier of at least one of a type of the device or atype of the imaging arrangement, in order to transmit said indication ofa disparity range.
 30. A method for receiving and processingthree-dimensional digital image data, comprising: receiving a first rawimage taken along a first optical axis, receiving a second raw imagetaken along a second optical axis; and receiving an indication of adisparity range between a maximum input disparity value and a minimuminput disparity value, wherein the maximum input disparity value is ameasure of a difference between a location in the first raw image thatrepresents a minimum imaging depth of an imaging arrangement used toproduce the first and second raw images and a location in the second rawimage that represents the minimum imaging depth, and the minimum inputdisparity value is a measure of a difference between a location in thefirst raw image that represents a maximum imaging depth of the imagingarrangement and a location in the second raw image that represents themaximum imaging depth.
 31. The method according to claim 30, wherein inorder to find locations for output subpixels to be displayed on anautostereographic display the method comprises linearly mapping inputdisparities in the first and second raw images that are between themaximum input disparity value and the minimum input disparity value intooutput disparities that are between a maximum output disparity value anda minimum output disparity value, of which the maximum output disparityvalue is a measure of a difference in location between two outputsubpixels that represent a virtual distance in front of a displayscreen, and the minimum output disparity value is a measure of adifference in location between two output subpixels that represent avirtual distance behind the display screen.
 32. The method according toclaim 31, wherein for displaying said output subpixels on anautostereographic display a default viewing distance of which is between20 and 60 centimeters, the method comprises using a maximum outputdisparity value and a minimum output disparity value the absolute valuesof which are between 2 and 10 millimeters.
 33. The method according toclaim 31, comprising changing the value of at least one of the maximumoutput disparity value and the minimum output disparity value inresponse to control inputs given by a user.
 34. A memory stored withinstructions such that when executed in a computer, causes the computerto: transmit a first raw image taken along a first optical axis;transmit a second raw image taken along a second optical axis; andtransmit an indication of a disparity range between a maximum disparityvalue and a minimum disparity value, wherein the maximum disparity valueis a measure of a difference between a location in the first raw imagethat represents a minimum imaging depth of an imaging arrangement usedto produce the first and second raw images and a location in the secondraw image that represents the minimum imaging depth, and the minimumdisparity value is a measure of a difference between a location in thefirst raw image that represents a maximum imaging depth of the imagingarrangement and a location in the second raw image that represents themaximum imaging depth.
 35. The memory stored with instructions accordingto claim 34, wherein the instructions when executed in a computer,causing the computer to transmit a maximum disparity in pixels between alocation in the first raw image that represents the minimum imagingdepth and a location in the second raw image that represents the minimumimaging depth, in order to transmit said indication of a disparityrange.
 36. The memory stored with instructions according to claim 35,wherein the instructions when executed in a computer, causing thecomputer to transmit a minimum disparity in pixels between a location inthe first raw image that represents the maximum imaging depth and alocation in the second raw image that represents the maximum imagingdepth.
 37. The memory stored with instructions according to claim 34,wherein the instructions when executed in a computer, causing thecomputer to transmit an identifier of at least one of a type of thedevice or a type of the imaging arrangement, in order to transmit saidindication of a disparity range.
 38. A memory stored with instructionssuch that when executed in a computer, cause the computer to: receive afirst raw image taken along a first optical axis; receive a second rawimage taken along a second optical axis; and receive an indication of adisparity range between a maximum input disparity value and a minimuminput disparity value, wherein the maximum input disparity value is ameasure of a difference between a location in the first raw image thatrepresents a minimum imaging depth of an imaging arrangement used toproduce the first and second raw images and a location in the second rawimage that represents the minimum imaging depth, and the minimum inputdisparity value is a measure of a difference between a location in thefirst raw image that represents a maximum imaging depth of the imagingarrangement and a location in the second raw image that represents themaximum imaging depth.
 39. The memory stored with instructions accordingto claim 38, wherein in order to find locations for output subpixels tobe displayed on an autostereographic display, wherein said instructionswhen executed in a computer, cause the computer to map input disparitiesin the first and second raw images that are between the maximum inputdisparity value and the minimum input disparity value into outputdisparities that are between a maximum output disparity value and aminimum output disparity value, of which the maximum output disparityvalue is a measure of a difference in location between two outputsubpixels that represent a virtual distance in front of a displayscreen, and the minimum output disparity value is a measure of adifference in location between two output subpixels that represent avirtual distance behind the display screen.
 40. The memory stored withinstructions according to claim 39, wherein for displaying said outputsubpixels on an autostereographic display a default viewing distance ofwhich is between 20 and 60 centimeters, wherein said instructions whenexecuted in a computer, cause the computer to use a maximum outputdisparity value and a minimum output disparity value the absolute valuesof which are between 2 and 10 millimeters.
 41. The memory stored withinstructions according to claim 39, wherein said instructions whenexecuted in a computer, cause the computer to change the value of atleast one of the maximum output disparity value and the minimum outputdisparity value in response to control inputs given by a user.
 42. Adevice comprising: means for taking a first raw image along a firstoptical axis and a second raw image along a second optical axis,wherein: the first and second optical axes have a separation at theimaging arrangement, the imaging arrangement has a maximum imaging depthand a minimum imaging depth; the device further comprising means fortransmitting the first raw image and the second raw image to a receivingdevice along with an indication of a disparity range between a maximumdisparity value and a minimum disparity value, the maximum disparityvalue is a measure of a difference between a location in the first rawimage that represents the minimum imaging depth and a location in thesecond raw image that represents the minimum imaging depth, and theminimum disparity value is a measure of a difference between a locationin the first raw image that represents the maximum imaging depth and alocation in the second raw image that represents the maximum imagingdepth.